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abstract: The use of a longitudinal database to study the impact of a uariety of student programs on student 
success mill be discussed . This study addresses the question of which programs and groups of programs most 
strongly influence student success. Programs designed to help engineering students at eight colleges are being 
studied in order to determine the factors that significantly contribute to student success. Types of programs being 
included in this study are mentoring programs, summer residential programs, engineering orientation programs, and 
others. Student success will be measured by graduation rates, grade point auerages, and time-to-graduation. 
although the primary study is still a work in progress, its potential to be a landmark study of this type make it 
desirable to discuss the research study at its present stage. 

a longitudinal database under continuing deuelopment by the NSF-sponsored Southeastern Uniuersity and College 
Coalition for Engineering Education (SUCCEED) will be used in this study. The database couers all entering cohorts 
from 1987 to the present. The longitudinal database is particularly useful for establishing control groups from the 
general population of engineering students, since data from all eight engineering colleges haue been placed in a 
common format. 

This work is sponsored by an NSE Postdoctoral Eellowship for Science, Mathematics, Engineering, and Technology 
Education (PESMETE) award. This paper will address the classification of programs into a smaller number of 
independent uariables that will allow comprehensiue programs to be broken down and their components analyzed 
separately. The approach to data collection and the statistical methods that will be used to quantify the relative 
benefit of different types of programs will also be discussed. The presentation of this paper will be interactive, as 
feedback from those attending the session will have an opportunity to affect the course of the study at this early 
stage. 

Keywords: longitudinal, assessment, success, retention, evaluation 



1 Introduction 

Chickering and Heisser (1993) identify friendships and student communities and student deuelopment programs and 
seruices as keg influences on student deuelopment. Hstin illustrates a wide uariety of effects of different types of 
student inuoluement (1993b). Tinto notes that effectiue retention programs are due to extra-curricular programs as 
much as or more than what occurs within the classroom (TINTO, 1987). Longitudinal studies by the Higher Education 
Research Institute and a wide uariety of other sources haue indicated that participation by students in uarious types 
of extra-curricular actiuities has a positiue influence on student persistence and success in their academic programs. 
(HSTIN and SHK, 1998, HSTIN, 1993a, 1993b, and 1984, MHLLETTE and CHBHEHH, 1991, NOHH, 1987, PHSCHRELLH and 
TEHENZINI, 1991 and 1 980, TEHENZINI and PHSCHRELLH, 1977, PHHKEH and SCHMIOT, 1982, OHV and BHHSKHMP, 1998, 
TINTO, 1998 and 1997, and OEV, 1997) 

This work, sponsored by an NSF Postdoctoral Fellowship for Science, Mathematics, Engineering, and Technology 
Education (PFSMETE) award, is proposed to extend what was learned in these and other longitudinal studies. The 
motiuation to collect information on student programs will be discussed, followed by a description of the SUCCEEO 
longitudinal database, which will be used in this study. If a broad and sufficiently comprehensiue set of programs 
that help students to be successful can be identified, a set of predictors can be identified through factor analysis 
(BOHG and GHLL, 1989, 620). H typical set of programs already identified will then be reuiewed. In order to reduce the 
number of independent uariables in the study, all programs included in the study will be broken down into 
components. The components that haue already been identified for classification will be discussed, although this set 
is still in a state of flux, as more programs are added to the set under consideration. By obseruing the uariance 
associated with uarious program components, the strength of the contribution of those components to student 
success will be approximated. Finally, the statistical approach to analysis of the program components will be 
included. 
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2 Motiuation to Collect Information on Student Programs 



This type of research is best performed through a longitudinal study (BORG and GRLL, 1989, p. 421, and RSTIN, 1991). 
The longitudinal database under deuelopment by the SUCCEED, described further below, indisputably contains a uast 
amount of data describing which students haue succeeded in the pursuit of engineering majors and which haue not. 
What the data alone do not tell us is luhy some students succeed and some do not. Uniuersities and their Colleges of 
Engineering offer a wide range of programs that are intended to positiuely influence student success. This study 
addresses the question of which program components most strongly influence student success. Types of programs 
being included in this study are mentoring programs, tutoring programs, summer residential programs, and 
engineering orientation programs. Other criteria affecting program inclusion are discussed later. Student success may 
be measured by persistence (retention), graduation rates, grade point auerages, and time-to-graduation. 

College and Uniuersity personnel implementing student programs must answer questions such as, "Does this program 
help students?" or "Does this program help students better than the program we used to use?" While the database 
used in this study can address these questions, this research plan intends to address the broader question, "Which 
program components and groups of program components most strongly influence student success?" Earlier 
longitudinal studies haue studied the effects of participation in seruice actiuities and other student programs on 
student success. The approach to identifying such student participation has always been through suruey instruments, 
because it has always been too intensiue to collect information directly from those that conduct student programs. 
The support of the NSF PFSMETE postdoctoral fellowship and the benefits of the SUCCEED partnership are helping to 
simplify the difficult task of collecting this kind of information. The additional benefit of this more comprehensiue 
approach is that, using information about the types of programs students participate in, different approaches to 
improuing student success can be weighed against one another in the content of determining how to most effectiuely 
help our students. The focus of this work on the success of engineering students allows the constraint of the 
programs under study. Since engineering majors are students, any programs open to all students are considered. 
Programs sponsored by the Colleges of Engineering are added to learn about the more comprehensiue set of 
programs auailable to engineering students. The much broader set of programs offered to students in other majors 
by colleges outside the College of Engineering is not considered. 



R longitudinal database is under continuing deuelopment by the NSF-sponsored Southeastern Uniuersity and College 
Coalition for Engineering Education (SUCCEED). The database contains data for all entering cohorts from 1987 to the 
present at SUCCEED member institutions. SUCCEEDS members include the headquarters at the Uniuersity of Florida (UF) 
and the other member institutions: Clemson Uniuersity, Florida ROM Uniuersity (FRMU), Florida State Uniuersity (FSU), 
Georgia Institute of Technology (Ga Tech), North Carolina ROT State Uniuersity (NC ROT), North Carolina State 
Uniuersity (NCSU), Uniuersity of North Carolina at Charlotte (UNCC), and Uirginia Polytechnic Institute and State 
Uniuersity (Ua Tech). These nine institutions contain eight engineering colleges, since FRMU and FSU haue a joint 
College of Engineering. The longitudinal database is particularly useful for establishing control groups from the 
general population of engineering students, since data from all eight engineering colleges haue been placed in a 
common format. 

Data are collected in a uariety of formats from the nine SUCCEED institutions. R description of each institution’s data 
definitions tempered with knowledge of the different academic policies allows merging of a common set of data in a 
consistent format. The SUCCEED longitudinal database has three components: 1) a demographic component, which 
contains unchanging data such as ethnicity, gender, matriculation date, matriculation major code, high school GPR, 

SRT scores, and RCT scores; 2) a term component, which contains data that change each term including term and 
cumulatiue GPR, major field of study, and course load; and 3) a graduation component, which contains records for 
each bachelor’s degree awarded to each student in the database. R common identifier, the student’s social security 
number, synchronizes records in each of the components of the database. The SUCCEED longitudinal database only 
includes records for undergraduate, degree-seeking students. R more complete description of the SUCCEED 
longitudinal database is auailable elsewhere (CRRSON, 1997). 

Another benefit of using the SUCCEED longitudinal database for this study is that it will continue to be augmented. All 
SUCCEED schools haue made a commitment to prouide supplementary data each year continuing until fiue years after 
the end of the current SUCCEED Cooperatiue Agreement in 2082. 



Rs discussed earlier, this study is limited to programs either offered by Colleges of Engineering or by the Uniuersities. 
Therefore, one criterion for inclusion is that a program either serues engineering majors specifically or all students 
generally. Since the SUCCEED longitudinal database only contains data for undergraduate, degree-seeking students, 
we can eliminate any programs that by their nature do not serue that population: recruitment and outreach 
programs, graduate programs, and continuing education programs aimed specifically at non-traditional students. 
Since the SUCCEED longitudinal database only has cohorts since 1987, programs that do not haue data for those 
cohortsare not of interest for this study. - — - ------ — - — 



3 The SUCCEED Longitudinal Database 



4 Criteria for Consideration in this Study 
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Rnother consideration is that programs included in this study mill generally be those programs that affect some 
sub-group of the population rather than the population as a whole. For example, all the institutions in this study haue 
some form of orientation during the summer prior to first-time-in-college matriculation. Since all (or uery nearly all) 
students attend these orientation sessions at their respectiue institutions, the orientation program becomes an 
integral part of the character of the institution. Measuring differences in the effects of different orientation 
programs will be seuerely confounded by other factors including the different populations that enroll at each 
institution and any differences in academic policies and record-keeping. This is because there is no control group of 
students at the same institution that haue not participated in those programs. Rll the orientation programs will, 
howeuer, be characterized to be sure that none is drastically different from the rest. Rlthough the effects of 
curricular changes ouer time will be difficult to identify, when major reuisions of special introductory engineering 
courses occur, they will be noted. This information will be of particular interest if any sustained improuements can be 
identified that were concomitant with the inception of the curricular change. 

Some programs will be identified but not included in the study, because indiuidual data are not auailable. In such 
cases, aggregate data of student participation will be obtained where possible in order to be aware of the impact on 
the study of the missing data. R number of programs haue already been identified for which data haue been kept, but 
are confidential. The creation of the SUCCEED longitudinal database and studies resulting from it haue been permitted 
on the basis of conducting educational research, one of the premises under which confidential student information 
may be giuen out. Unfortunately, programs of a counseling nature cannot be released under this exception. For 
example, the Thomas E. Cook Counseling Center at Uirginia Tech offers a uariety of success skills courses-and it 
would benefit the present study to follow the students who haue taken those courses. Students who come to the 
Cook Counseling Center, howeuer, do so with the understanding that their attendance in any euents at the Counseling 
Center are considered confidential, similar to any other uisit with a counselor (Rmenkhienan, 1 998). This 
confidentiality is couered by different regulations and cannot be uiolated for the purposes of educational research. 



The broad scope of this study precludes finding all possible programs in a single pass. Those aware of any student 
programs at NC State that are not mentioned here are asked to contact the authors. Typically, there is no one source 
to identify all auailable student resources at an institution. Rs a result, a wide range of sources will be used: 

• Searching Uniuersity LUorld-LJJjde Web pages 

• Campus uisits to speak with student program personnel 

• Secondary contacts recommended during meetings on first campus uisit 

• Student handbooks 

• Campus phone directories 

• Undergraduate course catalogs (student resources are frequently identified in such books) 

• Responses from those who read the lists here and recommend additions to the authors 

• Additions from those who read similar lists sent to personnel at each institution 

To date, campus uisits haue not been conducted at all the SUCCEED member institutions. Specifically, Clemson, Georgia 
Tech, NC R&T, NC State, and Uirginia Tech haue been uisited. The number of additional contacts identified during the 
first uisit to each campus clearly show that another uisit is indicated. Neuertheless, any programs that the reader 
might suggest for inclusion in this study are welcome. 

6 Example: Programs at North Carolina State Uniuersity 

First-Year Engineering Curriculum Reuision: R wide uariety of research into first-year engineering courses has been 
performed at NCSU in the first fiue years of SUCCEED (FELDER et al., 1997, BERUDOIN et a!., 1995, ZOROWSKI, 1996, 
SUTTON et al., 1995, PORTER and FULLER, 1998). The students in the experimental course cohorts will be identified, but 
are generally small in number when compared to the larger cohort of freshmen engineers. In preparation for the Fall 
1998 semester, a group of these freshman-year innouators and teachers gathered and discussed the characteristics 
desired in a freshman course and studied the couerage of those characteristics by the uarious approaches taken 
(PORTER, 1 998). This new course is being offered to a much larger cohort of 1188 first-year engineering students 
(PORTER et al., 1999, OLLIS et al., 1999, and VRRBROUGH et al., 1999). While not included specifically in the study of 
student programs analyzed by components, this significant curriculum reuision will be considered for its potential to 
change the baseline for student success. 

The First Year College (CONWRV, 1998) is a special program for students who haue not yet decided on a majorat NC 
State. In reality, this includes three populations-those who are truly undecided, those considering engineering but 
are hedging against the possibility of not being accepted for admission into the College of Engineering, and a third 
group who haue already been refused admission to the College of Engineering. The third population of these will not 
be admitted to the First Vear College starting in Fall 1999. Admission to the First Vear College has been subject to an 
academic index cutoff of 2.5 from 1995-1998 (admission below the cutoff is subject to special reuiew), and may be 
raised for the 1 999-2888 academic year. The First Vear College includes a uariety of features: 

• Weekly aduising by professional aduisors 



5 Student Program Identification 
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• H special course on topics of special interest to first year students 

• Scheduling of classes with others in the program 

• Faculty and peer mentoring and tutoring 

• Special residence halls for program participants 

The Rcademic Enrichment Center (STONEHOUSE, 1998) is a tutoring center that was opened by the First Vear College in 
Fall 1997. The Center was first open only open to those in the First Vear College and/or the Renewed Commitment 
Program (described below), but in Fall 1 998, it opened to all students. The center keeps track of the students who 
come to meet with its paid undergraduate mentors who are trained for the job. 

The Renewed Commitment Program (RCP) (STONEHOUSE, 1 998) addresses the special needs of students placed on 
academic warning (GPH below 2.8). RCP includes reflection groups that meet one hour per week in small groups with 
RCP staff members, indiuidual meetings with reflection group leaders, and at least one uisit to the Academic 
Enrichment Center per week. The program is uoluntary, but requires a signed commitment to participate in all 
program components. The program originated in 1995 in the Student Deuelopment Office at NCSU, where some earlier 
participant data resides. 

The Peer Mentor Program offered by the Rfrican-Rmerican Student Affairs Office (TURNER, 1998) began in the fall of 
1982 as a reorganization of a 1988 program in which Rfrican-Rmerican students were assigned to Rfrican-Hmerican 
faculty or staff members seruing as uolunteer mentors. In the early years, the program was small, but now serues 
well ouer half the Rfrican-Rmerican student population at NC State. Longitudinal data are auailable on program 
participants and mentors. 

The Rduocates for Minoritg Engineering Student Success (Rmes 2 ) (MITCHELL, 1998) offer a wide uariety of 
programs through the Minority Engineering Programs office. The Summer Transition Program (STP) is a residential, 
fiue-week program with academic coursework as well as special skills workshops. The comprehensiue description of 
the program's schedule found in each year's closing reports will help to quantify its components, which include study 
skills training, tutoring, study halls, social actiuities, and other actiuities. STRRT , a peer mentoring program that is an 
offshoot of STP, connects junior and senior students to entering freshmen, most (but not all) of whom were 
participants in STP. The STRRT program was introduced in Fall 1996 and includes explicit success skills training. Data on 
STP and STRRT participants are auailable. Information regarding other RMES 2 programs, including clustering, 
supplemental instruction, counseling, early interuention, a freshman orientation course, scholarships, and 
professional deuelopment are still being inuestigated. 

Women Engineers Networking Together (W.E.N.T.) (BQTTQMLEY, 1998) is a peer mentoring program that started in Fall 
1 998 and is run by the LUomen in Engineering Program and the Society of UJornen Engineers student chapter at NC 
State. The mentors and mentees meet at least once per week in some fashion, at least once per month face-to-face, 
and get together with all the other mentor/mentee at a larger function at least twice per semester. The 
approximately 148 mentor/mentee pairs are expected to maintain their relationship for one year. The UJornen in 
Engineering Program also offers an engineering sorority, uarious seminars and workshops, and a first-year weekend 
retreat for women engineering students. 

7 Student Program Component Classification 

Information about the structure of each student program will be kept in a separate database that can be merged 
with the student records of the SUCCEED longitudinal database on the basis of a program identifier. R number of 
important factors haue been identified, indicating a fairly complicated data structure: 

• The time per week spent on the program component 

• The incentiue for attending the program component 

• The group size that characterizes the program component 

• Rny selection bias for participation 

• The time in a student's course of study the program occurs 

• The relationship of the indiuiduals conducting the program to the student 

• The nature of the program (academic, social, etc.) 

• UJhether the program is residential 

Using this scheme, indiuidual programs will be classified. Comprehensiue programs will be broken down and their 
components analyzed separately-for example, the Renewed Commitment Program at NC State discussed earlier has 
multiple components in which each student in the program participates. In this case, the same program identifier will 
be used to label each of the components. In this way, when the program records are merged with the student 
records, a student participating in such a comprehensiue program will be identified as participating in each of its 
components. 

8 Data Analysis Strategy 



LUhen a single program is analyzed to determine its efficacy, there are many confounding factors. H common one is 
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that a particularly gifted faculty member or program coordinator can produce results that cannot be duplicated by 
others mho follow the same model. Hnother common problem is that selection biases can skew results because the 
control group is not truly comparable to the eHperimental group. This study is designed to minimize these two 
effects. Hnalyzing similar programs at multiple institutions giuen under different conditions by different program 
administrators eliminates the program coordinator effect. If any conclusions are borne out in this analysis, they may 
be less striking than the results indicated by the gifted coordinator, but the trends should be more robust. 

In the case of selection bias, matched-pair analysis, in which students in the control group and the eHperimental 
group are identified with ualues of certain independent uariables that are substantially the same. This technique 
eliminates the uariance of those independent uariables in order to focus on the contributions of the uariables under 
study (BDHG and GHLL, 1 989, p. 678). Hnother way in which a large number of independent uariables can be reduced to 
a smaller number of factors is through factor analysis (BDHG and GHLL, 1 989, p. 620). This technique can show how 
each of the program factors (the ways the programs are classified) contributes to student success. 

9 Conclusions 

If uarious program components such as hours spent with a peer mentor and hours spent on academic tutoring can be 
classified as to their impact on student success, a major contribution to the best practices in engineering education, 
and higher education in general, will be achieued. For enample, if the study were to show that two different 
approaches contributed to successful graduation in the same way, then schools could choose between them to 
achieue the same effect. It is easy to suggest that such a choice would be made based on cost. We suspect, howeuer, 
that other institutional objectiues may also guide such a choice. 

This work is just beginning. This paper addresses the classification of programs into a smaller number of independent 
uariables that will allow student programs to be broken down and their components analyzed separately. The 
approach to data collection and the statistical methods that will be used to quantify the relatiue benefit of different 
types of programs has also been discussed. The presentation of this paper will be highly interactiue, as feedback 
from those attending the session is desired to affect the course of the study at this early stage. 
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